{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Lamini Classifier Agent Toolkit (LCAT) Example Use\n",
    "First let's install the Lamini Python SDK."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pip install lamini --upgrade"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's import the Lamini Python SDK and set our API key. You can get your API key from [app.lamini.ai](https://app.lamini.ai/account) - every user gets $300 in free credits to get started.\n",
    "\n",
    "Checkout our [documentation](https://docs.lamini.ai/authenticate/#1-get-your-lamini-api-key) for other ways to store your api key."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import lamini\n",
    "\n",
    "#Uncomment the line below to add api key manually in-line.\n",
    "#lamini.api_key = \"<insert key from app.lamini.ai>\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we're all set up. The first thing we do is define a classifier - Lamini uses the concept of a \"project\" to contain one or many classifier versions for a given task.\n",
    "\n",
    "When you create a project, you define several key elements that can't be changed later (if we need to make changes, no sweat! creating a new classifier model takes just a few minutes):\n",
    "\n",
    "1. A project name\n",
    "2. A base model for the project to use\n",
    "3. A set of categories or classes or buckets that you like to place items into\n",
    "4. Descriptions of each category\n",
    "\n",
    "Projects also (eventually) contain one or many versions of:\n",
    "\n",
    "1. Example inputs and the expected output - we call this a dataset\n",
    "2. Models trained on the dataset(s) you've provided\n",
    "\n",
    "We will use add multiple datasets and create a few models as we iterate and build the best classifier we can!\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create a Project:\n",
    "\n",
    "Here we're creating a project named \"CustomerSupportTriageExample\" that will leverage the Meta-Llama-3.1-8B-Instruct model.\n",
    "\n",
    "> #### In this notebook we are going to be working through a sample problem of categorizing customer chat logs with an insurance company help desk. All the examples are made up. \n",
    "> #### We look forward to hearing how you apply the examples in this notebook to improve a process within your business! \n",
    "\n",
    "Let's Get Started!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from lamini.classify.lamini_classifier import LaminiClassifier\n",
    "import random\n",
    "import textwrap\n",
    "\n",
    "\n",
    "cls = LaminiClassifier(\n",
    "    f\"CustomerSupportTriageExample{random.randint(1000,9999)}\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Define the classes:\n",
    "Once the project is created, we define the classes. The more detailed the description, the higher your initial accuracy will be. It helps to give a few examples of keywords or phrases that are likely to appear in this category. You can think of the description as a prompt for Lamini to understand the definition.\n",
    "\n",
    "A template for how to structure the descriptions follows a typical prompting pattern:  \n",
    "\n",
    "- A natural language description. \n",
    "    - Think about this as how the category might be described to the customer support team in their Standard Operating Procedures (SOPs).  \n",
    "\n",
    "- Example 1  \n",
    "    \n",
    "- Example 2  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "classes = {\n",
    "    \"policy_details\": \"\"\"Chats about policy specifics like coverage, limits, and exclusions. \n",
    "    Two examples are: \n",
    "    1. What are the details of policy 123456?\n",
    "    2. Who are the dependents on my current policy?\n",
    "    \"\"\",\n",
    "    \"account_and_billing\": \"\"\"Chat messages asking about how the customer pays, or handling late payments. This could be about policies or other services. \n",
    "    For example: \n",
    "    1. What is my balance due this month?\n",
    "    2. I made a payment but I received an email that my balance was past due.\n",
    "    \"\"\",\n",
    "}\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Add Example(s) for each class\n",
    "\n",
    "Now that we have a description above for Lamini to use when creating the classifier, we can add additional tagged examples explicitly. These additional examples are training data. Adding example inputs is optional, but will help with accuracy. You can always do this later - we'll add more as a follow up step later in this notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "examples = {\n",
    "    \"policy_details\": [\n",
    "       r'''User: Hi, I'm considering purchasing an auto insurance policy from your company. Could you tell me what types of coverage are included?\n",
    "\n",
    "Bot: Of course! Our standard auto insurance policy includes liability coverage, collision coverage, comprehensive coverage, uninsured/underinsured motorist coverage, and personal injury protection. Is there a specific type of coverage you would like more details about?\n",
    "\n",
    "User: I'm particularly interested in understanding the comprehensive coverage. What does it entail?\n",
    "\n",
    "Bot: Comprehensive coverage helps cover damages to your vehicle that aren't caused by a collision, such as theft, vandalism, fire, or natural disasters. It also covers damage from falling objects or an animal striking your vehicle. Does this align with what you're looking for?\n",
    "\n",
    "User: yes thanks. Can I set up auto payment for this policy?''',\n",
    "r'''User: Hi, I'm considering getting a new insurance policy and wanted to understand more about the coverage options available.\n",
    "\n",
    "Bot: Hello! I'd be happy to help. Our policies typically cover areas such as personal injury, property damage, and liability. Is there a specific type of coverage you're interested in?\n",
    "\n",
    "User: I'm mainly interested in understanding the coverage for property damage. How extensive is it?\n",
    "\n",
    "Bot: For property damage, our policies cover a wide range of scenarios, including fire, theft, and natural disasters. You can choose different levels of coverage based on your needs. \n",
    "\n",
    "User: That's helpful. How do I determine the right level of coverage for my situation?\n",
    "\n",
    "Bot: Here is the link.\n",
    "\n",
    "User: Great! I saw the billing choices are limited and payment is always due the firts monday? Do you have auto pay?''',\n",
    "r'''User: Hi, I’m considering purchasing home insurance and would like to know more about the coverage options you offer.\n",
    "\n",
    "Bot: Hello! I'd be happy to help with that. Our home insurance policies can cover a range of incidents including natural disasters, theft, and accidental damage. Is there a specific coverage you are interested in?\n",
    "\n",
    "User: I’m interested in finding a policy that includes both theft and water damage due to plumbing issues. Is that possible?\n",
    "\n",
    "Bot: Yes, we do offer policies that include coverage for both theft and water damage caused by plumbing issues. Would you like more detailed information on what each of these covers?\n",
    "\n",
    "User: No. I want to know how much does it cost? What is the regular payment method?''',\n",
    "    ],\n",
    "    \"account_and_billing\": [\n",
    "        r'''User: Hi, I have a question about my insurance bill. Could you help me understand the charges for this month?\n",
    "\n",
    "Bot: Of course! Could you please provide your policy number so I can look up your billing details?\n",
    "\n",
    "User: Sure, my policy number is 12345678.\n",
    "\n",
    "Bot: Thank you. I see that this month's charges include your annual premium, which is divided into monthly payments. There's also a small fee for the additional roadside assistance coverage you added last month.\n",
    "\n",
    "User: Ah, I see now. Thank you for explaining that! Is there a way I can view this breakdown online as well?''',\n",
    "        r'''User: Hi, I'm trying to understand my recent insurance bill. Can you help me figure it out?\n",
    "\n",
    "Bot: Of course! I'm here to help. What specific part of the bill are you having trouble with?\n",
    "\n",
    "User: There's a section labeled \"\"adjustments\"\" with an additional charge. I don't understand what that means.\n",
    "\n",
    "Bot: Adjustments often account for changes in your policy, like a change in coverage or a modification in your payment plan. Would you like me to check if any specific modifications were made to your account?\n",
    "\n",
    "User: Yes, please. I made a change to my deductible last month, could that be it?''',\n",
    "        r''' User:  Hi, I have a question about my latest insurance bill. Could you help me figure out why it seems higher than usual?\n",
    "\n",
    " Bot:  Of course! I'd be happy to help. Could you provide your policy number so I can take a closer look at your billing details?\n",
    "\n",
    " User:  Sure, my policy number is 12345678.\n",
    "\n",
    " Bot:  Thank you! After reviewing your account, I see that there was an adjustment made this month due to a recent claim. This might explain the increase. Would you like more details on this adjustment?\n",
    "\n",
    " User:  Yes, please. I'd like to understand what the adjustment was for and if my bill will be lower next month.'''\n",
    "    ],\n",
    "}\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A simple function to print out the examples in a readable format. We will use this again when we add data programmatically a bit later.\n",
    "\n",
    "Lets review what we have so far."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Function to word-wrap and display\n",
    "def display_wrapped_content(data, width=200):\n",
    "    wrapper = textwrap.TextWrapper(width=width)\n",
    "    for key, conversations in data.items():\n",
    "        print(f\"Category: {key}\\n\" + \"=\"*len(f\"Category: {key}\"))\n",
    "        for i, conversation in enumerate(conversations, 1):\n",
    "            print(f\"\\nConversation {i}:\")\n",
    "            print(wrapper.fill(conversation))\n",
    "            print(\"\\n\" + \"-\"*width)\n",
    "            \n",
    "#Review the examples from our file\n",
    "display_wrapped_content(examples)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have our class descriptions and training examples, time to create the classifier. Use the **.initialize** endpoint to create a new project. This can take about a minute per class, so we'll put in a simple timer to keep us updated on status."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "resp = cls.initialize(classes, examples) \n",
    "\n",
    "import time\n",
    "\n",
    "while True:\n",
    "    print(\"Waiting for classifier to initialize\")\n",
    "    time.sleep(5)\n",
    "    resp = cls.train_status()\n",
    "    if resp[\"status\"] == \"completed\":\n",
    "        print(\"Model ID: \" + resp[\"model_id\"])\n",
    "        first_model_id = resp[\"model_id\"]\n",
    "        break\n",
    "    if resp[\"status\"] == \"failed\":\n",
    "        print(resp)\n",
    "        raise Exception(\"failed training\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Cool, we have our first model version! Let's try it out with a quick test."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 134,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\"classification\": [[{\"class_id\": 1, \"class_name\": \"account_and_billing\", \"prob\": 0.651953836198299}, {\"class_id\": 0, \"class_name\": \"policy_details\", \"prob\": 0.34804616380170095}]]}\n"
     ]
    }
   ],
   "source": [
    "import json\n",
    "\n",
    "# target: account_and_billing\n",
    "response = cls.classify(''' \n",
    "                        User:  Hi there, I have a question about my insurance bill. Can you help me with that?  \\n  \\n \n",
    "                        Bot:  Absolutely! I’d be happy to assist you with your billing question. What specifically would you like to know?  \\n  \\n \n",
    "                        User:  Could you tell me how I can view and pay my bill online?  \\n  \\n \n",
    "                        Bot:  Sure thing! You can view and pay your bill by logging into your account on our website. Once logged in, navigate to the \"Billing\" section where you\\'ll see options to view your statement and make payments using various methods like credit card or bank transfer.  \\n  \\n \n",
    "                        User:  Great, but I’m not sure I remember my login details. Can you guide me on how to reset my password?\n",
    "                        ''')\n",
    "\n",
    "\n",
    "print(json.dumps(response))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's take a quick look at the output above. We get a list of all the categories defined in our project, including a confidence score for each.\n",
    "\n",
    "We can go even further to easily quantify the accuracy of our classifier. Let's run an evaluation!\n",
    "\n",
    "What an evaluation means for a classifier: when you provide a set of inputs and the expected output, Lamini will test the accuracy of the model on those inputs, and give you back both overall metrics as well as per-input assessment. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Build Our Evaluation Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "# account_and_billing eval\n",
    "account_and_billing_eval = [\n",
    "    {\n",
    "        \"input\": \"User: Hi, I'm trying to figure out how to view my recent insurance bill online. Can you help me with that?\\n\\nBot: Certainly! To view your recent bill, you need to log into your account on our website. Once logged in, navigate to the \\\"Billing\\\" section where you can view and download your recent statements.\\n\\nUser: Okay, but I don't remember my login details. What should I do?\\n\\nBot: No worries! You can click on the \\\"Forgot Password\\\" link on the login page to reset your password. Just follow the instructions sent to your registered email to regain access.\\n\\nUser: Got it, thanks! Is there any option to set up automatic payments?\\n\\nBot: Yes, you can set up automatic payments from the \\\"Payment Settings\\\" in your account. This allows you to link a bank account or credit card for scheduled payments to ensure your bills are always paid on time.\\n\\nUser: Great, that sounds convenient. I'll try setting that up. Thanks for your help!\",\n",
    "        \"target\": \"account_and_billing\"\n",
    "    },\n",
    "    {\n",
    "        \"input\": \"User: Hi, I have a question about my upcoming insurance bill. When is it due?\\n\\nBot: Hello! Let me check that for you. Could you please provide your policy number?\\n\\nUser: Sure, it's 12345678.\\n\\nBot: Thank you! Your payment is due on the 15th of next month. Would you like to set up a reminder or schedule automatic payments?\\n\\nUser: No, that's okay. I just need to know if there are any late fees if I miss the due date.\",\n",
    "        \"target\": \"account_and_billing\"\n",
    "    },\n",
    "    {\n",
    "        \"input\": \"User: Hi, I need some help understanding my billing statement. Can you help me with that?\\n\\nBot: Of course! I'd be happy to help. What specifically would you like to know about your billing statement?\\n\\nUser: I'm not sure about the different charges on my bill. Could you explain what each charge is for?\\n\\nBot: Sure! Typically, your bill includes your monthly premium, any additional services or riders you have, and any applicable taxes or fees. Is there a particular charge you're curious about?\\n\\nUser: Yes, there's a charge labeled \\\"optional coverage fee\\\" that I'm not familiar with. Can you tell me more about it?\",\n",
    "        \"target\": \"account_and_billing\"\n",
    "    },\n",
    "    {\n",
    "        \"input\": \" User:  Hi, I have a question about my insurance bill. I noticed there's been a change in my premium this month. Can you help me understand why?\\n\\n Bot:  Certainly! Changes in premiums can occur due to a variety of reasons, such as policy updates, changes in coverage, or adjustments based on your risk profile. Would you like me to check your account specifics?\\n\\n User:  Yes, that would be great. I just want to confirm if it's due to some policy update or something else.\\n\\n Bot:  I've pulled up your account details, and it looks like the increase is due to a recent policy update focused on enhanced coverage. This update aims to provide more comprehensive benefits in the event of a claim.\\n\\n User:  Oh, I see. Thanks for clarifying that for me. Is there a way to get notified in advance about these changes?\",\n",
    "        \"target\": \"account_and_billing\"\n",
    "    },\n",
    "    {\n",
    "        \"input\": \"User: Hi, can you help me with understanding my insurance bill this month?\\n\\nBot: Of course! Could you please provide your policy number so I can look up the details?\\n\\nUser: Sure, my policy number is 123456789.\\n\\nBot: Thank you! I see your bill includes a premium payment and a small adjustment for a policy change last month. Would you like more details on these charges?\\n\\nUser: Yes, please explain the adjustment for the policy change.\",\n",
    "        \"target\": \"account_and_billing\"\n",
    "    },\n",
    "    {\n",
    "        \"input\": \"User: Hi there! I have a question about my billing statement. \\n\\nBot: Sure, I'd be happy to help! What specifically would you like to know about your billing statement?\\n\\nUser: I noticed an extra charge labeled \\\"service fee\\\" on my last bill. What is this fee for?\\n\\nBot: The service fee is typically applied for administrative costs related to managing your policy. However, it shouldn't be unexpected. Would you like me to check if it's a recurring charge on your plan?\\n\\nUser: Yes, please! I'd appreciate that.\",\n",
    "        \"target\": \"account_and_billing\"\n",
    "    },\n",
    "    {\n",
    "        \"input\": \"User: Hi, I'm trying to understand why my premium this month seems higher than usual. Can you help with that?\\n\\nBot: Of course! There could be a few reasons for an increase in your premium. Have you recently updated your policy or added any new coverage?\\n\\nUser: Hmm, I don't think I've made any changes. Could it be something else?\\n\\nBot: It's possible that there were changes in your area that affected your rate or adjustments made at the time of your policy renewal. I can also check if any discounts have expired.\\n\\nUser: That would be great! Could you take a look at whether any discounts I had have been removed?\",\n",
    "        \"target\": \"account_and_billing\"\n",
    "    }\n",
    "]\n",
    "\n",
    "# policy_details eval\n",
    "policy_details_eval = [\n",
    "    {\n",
    "        \"input\": \"User: Hi, I'm interested in knowing more about the coverage details of your home insurance policy. Can you help me with that?\\n\\nBot: Of course! Our home insurance policy typically covers damages caused by natural disasters, theft, and vandalism. Would you like information on any specific coverage?\\n\\nUser: That sounds great. Can you also tell me how long the coverage is for once I purchase the policy?\\n\\nBot: Yes, the standard home insurance policy is usually valid for one year from the date of purchase. You have the option to renew annually. \\n\\nUser: Perfect, thank you! I'll think about it and possibly reach out for more details.\",\n",
    "        \"target\": \"policy_details\"\n",
    "    },\n",
    "    {\n",
    "        \"input\": \"User: Hi, I was wondering if you could tell me about the coverage options available for a comprehensive car insurance policy.\\n\\nBot: Of course! Our comprehensive car insurance covers a range of incidents, including theft, vandalism, natural disasters, and accidents. It also includes liability coverage for damages to other vehicles or property.\\n\\nUser: Thanks! How long does a typical policy term last for car insurance?\\n\\nBot: A standard car insurance policy term is usually 12 months, but we also offer six-month policies if you prefer shorter terms.\\n\\nUser: That's good to know. Can I make adjustments to the coverage during the policy term, or do I need to wait until renewal?\",\n",
    "        \"target\": \"policy_details\"\n",
    "    },\n",
    "    {\n",
    "        \"input\": \"User: Hi, I was wondering if you could give me more details about what kind of coverage your auto insurance policy offers?\\n\\nBot: Absolutely! Our auto insurance policies typically cover liability, collision, comprehensive, personal injury protection, and uninsured motorist coverage. Is there a specific part of the coverage you're interested in?\\n\\nUser: I'm particularly interested in the comprehensive coverage. Can you tell me more about what's included in that?\\n\\nBot: Certainly! Comprehensive coverage helps protect your vehicle against non-collision incidents like theft, vandalism, natural disasters, and falling objects. It’s a great way to have broader protection for your car.\\n\\nUser: That sounds useful! Could you also let me know how long the policy usually lasts?\",\n",
    "        \"target\": \"policy_details\"\n",
    "    },\n",
    "    {\n",
    "        \"input\": \"User: Hi, I was wondering if you could provide me with some details about the coverage offered in your standard homeowner's insurance policy.\\n\\nBot: Of course! Our standard homeowner's insurance policy typically covers damages to the structure of your home, personal belongings, liability protection, and additional living expenses if your home becomes uninhabitable due to covered damages.\\n\\nUser: That sounds comprehensive. Can you tell me how long the coverage period usually lasts?\\n\\nBot: Sure! Our coverage period is typically for one year, but you can renew it annually. We also offer options for multi-year policies if you're interested in extended coverage.\\n\\nUser: Great, thank you for the information. Is there a way to customize the coverage within that period if I need to make adjustments later on?\",\n",
    "        \"target\": \"policy_details\"\n",
    "    },\n",
    "    {\n",
    "        \"input\": \"User: Hi, I'm considering getting a health insurance policy and wanted to know what types of coverage are typically included?\\n\\nBot: Hello! I'd be happy to help with that. Our standard health insurance policies usually cover doctor visits, hospital stays, prescription medications, and emergency services. Do you need details on additional coverage options?\\n\\nUser: Yes, that would be great. Are there options for adding dental and vision coverage?\\n\\nBot: Absolutely, you can add dental and vision coverage to your plan. These typically include regular check-ups, eye exams, and certain corrective procedures. Would you like more information on specific plans for those?\\n\\nUser: That sounds good. Could you also let me know the typical length of a policy term?\",\n",
    "        \"target\": \"policy_details\"\n",
    "    },\n",
    "    {\n",
    "        \"input\": \"User: Hi, I have a question about my insurance payment. Can you help me understand how I can pay my premium online?\\n\\nBot: Of course! You can pay your premium online through our website or mobile app. Just log in to your account, navigate to the \\\"Billing\\\" section, and select \\\"Make a Payment.\\\"\\n\\nUser: Great, but what payment methods do you accept online?\\n\\nBot: We accept various payment methods online, including credit cards, debit cards, and e-checks. You can choose your preferred option during the payment process.\\n\\nUser: Is there a way to set up automatic payments so I don't forget next time?\\n\\nBot: Absolutely! You can set up automatic payments through the same \\\"Billing\\\" section. Just opt for the \\\"Auto-Pay\\\" feature and follow the steps to link your preferred payment method.\\n\\nUser: Thanks for the information! I'll set that up right away. Additionally, any new policy to introduce?\",\n",
    "        \"target\": \"policy_details\"\n",
    "    },\n",
    "    {\n",
    "        \"input\": \"User: Hi, I'm trying to figure out how to make a payment for my insurance premium. Can you guide me?\\n\\nBot: Of course! You can make a payment through our online portal, via phone, or by mailing a check. Would you like more details on any of these options?\\n\\nUser: Yes, I'd like more information about using the online portal.\\n\\nBot: Sure! To use the online portal, you need to log into your account on our website. Once logged in, navigate to the \\\"Billing & Payments\\\" section, and you can pay with a credit card, debit card, or via bank transfer.\\n\\nUser: Great, I think I can manage that. Meanwhile, any fun detail about the new policy I shall know? \",\n",
    "        \"target\": \"policy_details\"\n",
    "    }\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Run the evaluation\n",
    "\n",
    "Using the evaluation endpoint, we simply pass the eval dataset we just built to the **eval_data** parameter of the Lamini Evaler."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from lamini.one_evaler.one_evaler import LaminiOneEvaler\n",
    "\n",
    "eval = LaminiOneEvaler(\n",
    "    test_model_id=first_model_id,\n",
    "    eval_data_id=f\"first_eval{random.randint(1000,9999)}\",\n",
    "    eval_data=account_and_billing_eval+policy_details_eval,\n",
    "    test_eval_type=\"classifier\",\n",
    ")\n",
    "\n",
    "full_eval = eval.run()\n",
    "\n",
    "print(json.dumps(full_eval, indent=2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Even with just a few examples of each class, the model already got 85% of eval inputs correct. This is captured in the **tuned_accuracy** field of the eval output.\n",
    "\n",
    ">Best practice: after you've added a few high-quality examples, you should run an eval and carefully review the ground truth labels to make sure they're aligned with the classifier's task and scope - the tags in a general eval set aren't always the best for a narrowly-defined classifier agent to learn from. Don't just review inputs where the assigned class was completely wrong - also review inputs where the classifier's answer is correct but the confidence score is low.\n",
    "\n",
    "> Reminder: We can view the confidence by using **cls.classify** like we did above.\n",
    "\n",
    "Lets take a look at which examples the classifier missed. We can use the below function to print out and view the eval examples that the model got wrong."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def print_missed_evals(eval):\n",
    "    missed = [\n",
    "        pred for pred in eval['predictions'] \n",
    "        if pred['test_output'] != pred['target']\n",
    "    ]\n",
    "    \n",
    "    if missed:\n",
    "        print(\"Missed evals:\")\n",
    "        for prediction in missed:\n",
    "            print(f\"Input: {prediction['input']}\")\n",
    "            print(f\"Expected: {prediction['target']}\")\n",
    "            print(f\"Predicted: {prediction['test_output']}\")\n",
    "            print(\"-\" * 80)\n",
    "    else:\n",
    "        print(\"No Missed Evals!\")     \n",
    "\n",
    "print_missed_evals(full_eval)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### What should we do to get higher accuracy?\n",
    "\n",
    "To Start: More, well curated training data typically helps, lets add a few more training examples. We only added a few for each class so far."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Load Training Examples from a file:\n",
    "\n",
    "Most times our data is likely going to be ingested in an unstructured format from our chat application. Let's take a look at how we might stage that data into the right format to be used to train the classifier. \n",
    "\n",
    "> Note: This is additional training data, examples we use here should be ideal examples that our Classifier agent is going to learn from.\n",
    "\n",
    "The below snippet, loads data from a file that is in the data directory of this repo. It is pre-populated with some data for us. As we run the below code, it will print out the loaded examples to review briefly before we train the classifier again (**.initialize** includes the first training run)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "import os\n",
    "\n",
    "# verify the location of where our examples are stored, there are a few samples here for you already\n",
    "filepath_with_examples = \"./data/insurance_convos.jsonl\"\n",
    "\n",
    "def load_jsonl_file(relpath):\n",
    "    data = []\n",
    "    \n",
    "    #get the current director where the notebook is running\n",
    "    current_dir = os.path.dirname(os.getcwd())\n",
    "    \n",
    "    #add the relative path to the data directory with our filename\n",
    "    file_path = os.path.join(current_dir, relpath)\n",
    "    \n",
    "    #load data and return the examples\n",
    "    with open(file_path, 'r') as file:\n",
    "        for line in file:\n",
    "            data.append(json.loads(line))\n",
    "    return data\n",
    "\n",
    "def write_jsonl_file(filepath, data):\n",
    "    with open(filepath, 'a') as file:\n",
    "        for entry in data:\n",
    "            file.write(json.dumps(entry) + \"\\n\")\n",
    "\n",
    "def format_examples_from_file(data):\n",
    "    examples = {\n",
    "            \"policy_details\": [\n",
    "                line['conversation'] for line in data if line['category'] == \"policy_details\"\n",
    "            ],\n",
    "            \"account_and_billing\": [\n",
    "                line['conversation'] for line in data if line['category'] == \"account_and_billing\"\n",
    "            ],\n",
    "        }\n",
    "    return examples\n",
    "\n",
    "#load data from jsonl file into array of items\n",
    "loaded_data = load_jsonl_file(filepath_with_examples)\n",
    "\n",
    "#parse the array into the format required by the Lamini examples endpoint\n",
    "examples_from_file = format_examples_from_file(loaded_data)\n",
    "\n",
    "#Review the examples from our file\n",
    "display_wrapped_content(examples_from_file)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Review Programmatically Loaded Examples Above\n",
    "\n",
    "Our new examples look good!\n",
    "\n",
    "Last time we added training examples, we used examples straight away when creating the classifier. Here we are going to add new examples to the same classifier to provide additional data. We will use the same **cls** object that we created already and call the **.add** method. Passing in a unique name for this dataset and the examples. \n",
    "\n",
    "### Train the Classifier with the New Examples\n",
    "\n",
    "Because we are adding additional data explicitely, we need to call **.train** when we have added all the data we want to add to this next model of our classifier project. \n",
    "> Note: We can add more than one dataset before training. Training is fast, typically each iteration can be 1 dataset as we iterate on improving the classifier."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "resp = cls.add(\n",
    "    f\"additional_data{random.randint(1000,9999)}\", examples_from_file\n",
    ")\n",
    "\n",
    "resp = cls.train()\n",
    "\n",
    "while True:\n",
    "    print(\"Waiting for classifier to train\")\n",
    "    time.sleep(5)\n",
    "    resp = cls.train_status()\n",
    "    if resp[\"status\"] == \"completed\":\n",
    "        print(\"Model ID: \" + resp[\"model_id\"])\n",
    "        second_model_id = resp[\"model_id\"]\n",
    "        break\n",
    "    if resp[\"status\"] == \"failed\":\n",
    "        print(resp[\"status\"])\n",
    "        raise Exception(\"failed training\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Great, now we have a second model version in our project! Let's run an eval and compare it to the first version. \n",
    "\n",
    "> #### Note: When comparing 2 models, you'll notice we pass a few extra parameters to the Evaler. The test model continues the be the model that we want to understand its behavior (our most recent one). The model we are comparing to, is the base model. \n",
    "\n",
    "### Lets see the results from the updated classifier!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Running comparison eval between model versions \" + first_model_id + \" and \" + second_model_id)\n",
    "\n",
    "eval_2 = LaminiOneEvaler(\n",
    "    test_model_id=second_model_id,\n",
    "    eval_data_id=f\"second_eval{random.randint(1000,9999)}\",\n",
    "    eval_data=account_and_billing_eval+policy_details_eval,\n",
    "    test_eval_type=\"classifier\",\n",
    "    base_model_id=first_model_id,\n",
    "    sbs=True,\n",
    "    fuzzy=True,\n",
    ")\n",
    "\n",
    "full_eval2 = eval_2.run()\n",
    "\n",
    "print(json.dumps(full_eval2, indent=2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Which eval did the model miss?\n",
    "\n",
    "The eval output makes it easy to compare model versions overall, and to see exactly where the differences are, so you know exactly where to focus to improve your workflow.\n",
    "\n",
    "Lets take a look at all of the eval data points above that the classifier did not get correct."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print_missed_evals(full_eval2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So our model missed the same two...\n",
    "\n",
    "### How do we think about getting even higher accuracy!?\n",
    "\n",
    "1. We should look at the content of the missed input(s), closely\n",
    "2. Source or generate examples that cover the relevant concepts or structure of the missed input(s)\n",
    "3. Add them to the classifier as training examples\n",
    "4. Train\n",
    "5. Verify with Eval"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Its Your Lucky Day, I took a close look at the above examples for us\n",
    "\n",
    "> #### If you notice the most recent User response in the chat samples, its subtle at first ... The user changes topics well into the conversation. Sometimes even after the initial conversation has concluded. Ideally, we can have our chat experience identify this and continue the conversation with a relevant response. \n",
    "\n",
    "Using the same functions we defined above, lets print out our new dataset that is targeted to help fix this. Aptly named 'same_as_before'_v2 of course. It's also provided in the data directory of this repo."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# separate file with our new examples\n",
    "filepath_with_examples = \"./data/insurance_convos_v2.jsonl\"\n",
    "\n",
    "#load data from jsonl file into array of items\n",
    "loaded_data = load_jsonl_file(filepath_with_examples)\n",
    "\n",
    "#parse the array into the format required by the Lamini examples endpoint\n",
    "examples_from_V2_file = format_examples_from_file(loaded_data)\n",
    "\n",
    "#Review the examples from our file\n",
    "display_wrapped_content(examples_from_V2_file)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### We use the same flow as before, lets add our dataset and then train our model with the updated examples"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "resp = cls.add(\n",
    "    f\"additional_data{random.randint(1000,9999)}\", examples_from_V2_file\n",
    ")\n",
    "\n",
    "resp = cls.train()\n",
    "\n",
    "while True:\n",
    "    print(\"Waiting for classifier to train\")\n",
    "    time.sleep(5)\n",
    "    resp = cls.train_status()\n",
    "    if resp[\"status\"] == \"completed\":\n",
    "        print(\"Model ID: \" + resp[\"model_id\"])\n",
    "        third_model_id = resp[\"model_id\"]\n",
    "        break\n",
    "    if resp[\"status\"] == \"failed\":\n",
    "        print(resp[\"status\"])\n",
    "        raise Exception(\"failed training\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### As you guessed, now we run eval to see the models performance side by side\n",
    "\n",
    "This time between model version 2 and the current version 3."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Running comparison eval between model versions \" + second_model_id + \" and \" + third_model_id)\n",
    "\n",
    "eval_3 = LaminiOneEvaler(\n",
    "    test_model_id=third_model_id,\n",
    "    eval_data_id=f\"second_eval{random.randint(1000,9999)}\",\n",
    "    eval_data=account_and_billing_eval+policy_details_eval,\n",
    "    test_eval_type=\"classifier\",\n",
    "    base_model_id=second_model_id,\n",
    "    sbs=True,\n",
    "    fuzzy=True,\n",
    ")\n",
    "\n",
    "full_eval3 = eval_3.run()\n",
    "\n",
    "print(json.dumps(full_eval3, indent=2))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print_missed_evals(full_eval3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Excellent! \n",
    "### Our targeted new examples were able to help the model categorize a tricky change in topic correctly now! Great Job!\n",
    "\n",
    "> ### **Challenge**: Can you add a few more examples to get the model to 100%. Share your training data and results with us to receive an extra $100 in credits on app.lamini.ai\n",
    "\n",
    "# Happy Classifying!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "lamini",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
